Electronic apparatus for object recognition and control method thereof

ABSTRACT

An electronic apparatus is disclosed. The electronic apparatus includes a sensor, a camera, a memory, a camera and a processor. The memory stores a plurality of artificial intelligence models trained to identify objects and stores information on a map. The first processor provides, to the second processor, area information on an area in which the electronic apparatus is determined, based on sensing data obtained from the sensor, to be located, from among a plurality of areas included in the map. The second processor loads at least one artificial intelligence model of the plurality of artificial intelligence models to the volatile memory based on the area information and identifies an object by inputting the image obtained through the camera to the loaded artificial intelligence model.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of U.S. patentapplication Ser. No. 16/890,337, filed Jun. 2, 2020, which is based onand claims priority under 35 U.S.C. § 119 to Korean patent applicationnumber 10-2019-0065519, filed on Jun. 3, 2019, Korean patent applicationnumber 10-2019-0099904, filed on Aug. 14, 2019, and Korean patentapplication number 10-2019-0122580, filed on Oct. 2, 2019, in the KoreanIntellectual Property Office, the disclosure of which is incorporated byreference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an electronic apparatus for objectrecognition. More specifically, the disclosure relates an electronicapparatus configured to perform object recognition using an artificialintelligence model corresponding to a location of the electronicapparatus.

2. Description of Related Art

Recently, with artificial intelligence models being used in objectrecognition technology, object recognition capability of electronicapparatuses has greatly improved.

However, as more developed forms of artificial intelligence models areused to more accurately recognize far more objects, the amount ofprocessing performed has proportionately increased and electronicapparatuses require a very large memory capacity or memory speed.

Accordingly, there have been limitations in overcoming the objectrecognition capability of user devices, specifically with respect tomemory, processing capability and communication capability, by simplyimproving only the functions of the artificial intelligence modelitself.

SUMMARY

The disclosure provides an electronic apparatus configured to performobject recognition using an artificial intelligence model correspondingto a location of the electronic apparatus, and a method of operating thesame.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, an electronic apparatusincludes: a sensor; a camera; a storage for storing a plurality ofartificial intelligence models trained to identify objects and forstoring information on a map; a first processor configured to controlthe electronic apparatus; and a second processor configured to recognizean object, wherein the first processor is configured to determine, basedon sensing data obtained from the sensor, an area in which theelectronic apparatus is located from among a plurality of areas includedin the map, and provide area information on the determined area to thesecond processor, and wherein the second processor includes a volatilememory and is configured to: load an artificial intelligence model, fromamong the plurality of artificial intelligence models stored in thestorage, to the volatile memory based on the area information providedby the first processor, and input an image obtained through the camerato the loaded artificial intelligence model to identify an object.

In accordance with another aspect of the disclosure, an electronicapparatus includes: a camera; a sensor; a storage for storing aplurality of artificial intelligence models trained to identify objectsand for storing information on a map; and a processor configured tocontrol the electronic apparatus, wherein the processor includes avolatile memory, and wherein the processor is configured to: determine,based on sensing data obtained from the sensor, an area in which theelectronic apparatus is located from among a plurality of areas includedin the map, based on the determined area, load an artificialintelligence model from among the plurality of artificial intelligencemodels stored in the storage, and input an image obtained through thecamera to the loaded artificial intelligence model to identify anobject.

In accordance with another aspect of the disclosure, a control method ofan electronic apparatus using an object recognition model includes:identifying a plurality of areas included in a map based on informationon the map stored in a storage of the electronic apparatus; determiningan area in which the electronic apparatus is located from among theplurality of areas based on sensing data obtained from a sensor;loading, to a volatile memory, an artificial intelligence model, fromamong a plurality of artificial intelligence models stored in thestorage, based on the determined area; and identifying an object byinputting an image obtained through a camera to the loaded artificialintelligence model.

In accordance with another aspect of the disclosure, an electronicapparatus includes: a storage; and at least one processor including avolatile memory and configured to: determine an area in which theelectronic apparatus is located from among a plurality of areas includedin a map, based on the determined area, load an artificial intelligencemodel, from among the plurality of artificial intelligence models storedin the storage, corresponding to the determined area, and input an imageobtained through the camera to the loaded artificial intelligence modelto identify an object.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description, taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram illustrating an example of an electronic apparatusidentifying an object using different artificial intelligence models foreach area (e.g., living room and kitchen) according to an embodiment;

FIG. 2 is a block diagram illustrating a configuration of an electronicapparatus according to an embodiment;

FIG. 3A is a diagram illustrating an embodiment of an electronicapparatus determining an area where an electronic apparatus is locatedfrom a plurality of areas;

FIG. 3B is a diagram illustrating an embodiment of an electronicapparatus determining an area where an electronic apparatus is locatedfrom among a plurality of areas;

FIG. 4 is a diagram of a table illustrating an example of a plurality ofartificial intelligence models stored in a storage of an electronicapparatus;

FIG. 5 is a diagram illustrating a specific example of selectivelyloading an artificial intelligence model corresponding to a determinedarea from among a plurality of artificial intelligence models consistingof a convolutional layer and a fully-connected layer;

FIG. 6A is a diagram illustrating an example of an electronic apparatusgenerating a first dimensional map through a LiDAR sensor;

FIG. 6B is a diagram illustrating an example of an electronic apparatusdividing a first dimensional map to a plurality of areas using resultsidentifying a wall, a door, or the like;

FIG. 6C is a diagram illustrating an example of an electronic devicerecognizing objects in each divided area to identify use in each of theidentified plurality of areas;

FIG. 6D is a diagram illustrating an example of an electronic apparatusidentifying purpose of each area of the divided plurality of areasthrough communication with an external apparatus;

FIG. 7A is diagram illustrating an example of an electronic apparatusobtaining an artificial intelligence model corresponding to each of aplurality of areas using an object located in each of a plurality ofareas;

FIG. 7B is diagram illustrating an example of an electronic apparatusobtaining an artificial intelligence model corresponding to each of aplurality of areas using an object located in each of a plurality ofareas;

FIG. 7C is diagram illustrating an example of an electronic apparatusobtaining an artificial intelligence model corresponding to each of aplurality of areas using an object located in each of a plurality ofareas;

FIG. 8A is a diagram illustrating an example of an electronic apparatusupdating an artificial intelligence model corresponding to a relatedarea in case an object located at one area of a plurality of areas isidentified as not being located at the related area any longer;

FIG. 8B is a diagram illustrating an example of an electronic apparatusupdating an artificial intelligence model corresponding to a relatedarea in case an object located at one area of a plurality of areas isidentified as not being located at the related area any longer;

FIG. 8C is a diagram illustrating an example of an electronic apparatusupdating an artificial intelligence model corresponding to a relatedarea in case an object located at one area of a plurality of areas isidentified as not being located at the related area any longer;

FIG. 9A is a diagram illustrating an example of an electronic apparatusupdating an artificial intelligence model corresponding to a relatedarea in case a new object is identified as being added to one area of aplurality of areas;

FIG. 9B is a diagram illustrating an example of an electronic apparatusupdating an artificial intelligence model corresponding to a relatedarea in case a new object is identified as being added to one area of aplurality of areas;

FIG. 9C is a diagram illustrating an example of an electronic apparatusupdating an artificial intelligence model corresponding to a relatedarea in case a new object is identified as being added to one area of aplurality of areas;

FIG. 10 is a block diagram illustrating a configuration of an electronicapparatus according to various embodiments;

FIG. 11 is a diagram illustrating various embodiments of an electronicapparatus performing object recognition based on communication withexternal apparatuses including a server apparatus and an externalterminal apparatus;

FIG. 12A is a block diagram illustrating a configuration of anelectronic apparatus including a processor;

FIG. 12B is a block diagram illustrating a configuration of anelectronic apparatus including a processor;

FIG. 13 is a flowchart illustrating a control method of an electronicapparatus according to an embodiment;

FIG. 14 is a flowchart illustrating an embodiment of a control method ofan electronic apparatus according to an embodiment generatinginformation on a map, and identifying objects present in each of aplurality of areas to obtain an artificial intelligence modelcorresponding to each of the plurality of areas; and

FIG. 15 is a diagram of an algorithm illustrating an example of acontrol method of an electronic apparatus according to an embodimentupdating an artificial intelligence model corresponding to each of aplurality of areas according to results identifying an object present ineach of the plurality of areas.

DETAILED DESCRIPTION

The disclosure relates to an electronic apparatus performing highlyefficient object recognition by selectively using onlycircumstance-appropriate artificial intelligence models.

More specifically, the disclosure provides an electronic apparatuscapable of recognizing objects of various types, while being able torecognize objects relatively quickly compared to its limited processingcapabilities.

Before describing embodiments below, the description method of thedisclosure and drawings are described.

First, the terms used in the disclosure are general terms identified inconsideration of the functions of the various embodiments of thedisclosure. However, these terms may vary depending on intention, legalor technical interpretation, emergence of new technologies, and the likeof those skilled in the related art. In addition, there may be somearbitrary terms selected by an applicant. The terms may be interpretedbased on the meanings as defined herein, and may be construed based onthe overall content of the disclosure and the technological common senseof those skilled in the related art in case no specific definition ofthe term is described.

In addition, like reference numerals or characters disclosed in each ofthe drawings attached herein indicate a component or element performingsubstantially the same functions. For convenience of descriptions andunderstandings, like reference numerals or characters may be used todescribe different embodiments. That is, even if an element with a likereference numeral is illustrated in all of a plurality of drawings, theplurality of drawings may not necessarily refer to only one embodiment.

In addition, the terms including ordinal numbers such as “first” and“second” may be used to differentiate between elements in thedisclosure. These ordinal numbers are used merely to distinguish same orsimilar elements from one another, and should not be understood aslimiting the meaning of the terms as a result of using these ordinalnumbers. For example, the elements associated with the ordinal numbersshould not be limited in order or order of use by the numbers. Ifnecessary, each ordinal number may be used interchangeably.

A singular expression in the disclosure includes a plural expression,unless otherwise specified clearly in context. It is to be understoodthat the terms such as “comprise” or “consist of” are used herein todesignate a presence of a characteristic, a number, a step, anoperation, an element, a component, or a combination thereof, and not topreclude a presence or a possibility of adding one or more of othercharacteristics, numbers, steps, operations, elements, components or acombination thereof.

In the description, terms such as “module,” “unit,” and “part” may beused to refer to an element performing at least one function oroperation, and these elements may be implemented as hardware orsoftware, or a combination of hardware and software. Further, except forwhen each of a plurality of “modules,” “units,” “parts,” and the likeneeds to be implemented in an individual hardware, the components may beintegrated in at least one module or chip and be implemented in at leastone processor

In addition, in an embodiment, when any part is indicated as connectedto another part, this includes not only a direct connection, but also anindirect connection through another medium. Further, when a particularpart includes a particular element, an another element may be furtherincluded rather than precluding the other element, unless otherwisespecified.

FIG. 1 is a diagram illustrating an example of an electronic apparatusidentifying an object using different artificial intelligence models foreach area (e.g., living room and kitchen) according to an embodiment.

FIG. 1 illustrates an object recognition process of an electronicapparatus 10 implemented as a robot cleaner according to an embodiment.FIG. 1 illustrates a situation where a plurality of artificialintelligence models for recognizing various objects such as a television(TV), a sofa, a bed, a closet, clothes, foreign substance, a chair, asink, and an air conditioner is stored in a non-volatile memory or astorage of an electronic apparatus 10.

Referring to FIG. 1, the electronic apparatus 10, which is a robotcleaner, may move around areas 1-1, 1-2, and 1-3 on a map 1photographing images of various objects, and may input the photographedimages to a plurality of artificial intelligence models to recognizeobjects within an image.

In this case, a capacity or processing rate of a volatile memory of theelectronic apparatus 10 may be insufficient for the electronic apparatus10 to load the plurality of artificial intelligence models stored in thenon-volatile memory such as a storage and to perform object recognition.

Accordingly, the electronic device 10 may determine an area in which theelectronic apparatus 10 is located from the areas on the map 1, andload, to a volatile memory, only the artificial intelligence modelcorresponding to the area in which the electronic apparatus 10 islocated from among the stored plurality of artificial intelligencemodels to recognize the object in the image of the photographed object.

To this end, each of the plurality of artificial intelligence modelsstored for different areas may be pre-stored in the electronic apparatus10.

For example, referring to table (2) in FIG. 1, a living room model 2-1may recognize an air conditioner, a TV, and the like, which are normallylocated in the living room. On the other hand, the living room model 2-1may not recognize a refrigerator, a bed, and the like not normallylocated in the living room.

In addition, referring to table (2), a bedroom model 2-2 may notrecognize a refrigerator while recognizing an air conditioner, a TV, abed, and the like.

Accordingly, if the electronic apparatus 10 is located in the livingroom 1-1, the electronic apparatus 10 may selectively load, to thevolatile memory of a processor, only the living room model 2-1, fromamong the stored plurality of artificial intelligence models, torecognize the air conditioner, the TV, and the like, and use the loadedliving room model 2-1 to perform recognition on the object in the imagephotographed from the living room 1-1.

On the other hand, if the electronic apparatus 10 is located in a bedroom 1-2, the electronic apparatus 10 may selectively load, to thevolatile memory of a processor, only the bed room model 2-2, from amongthe stored plurality of artificial intelligence models, to recognize theair conditioner, the TV, the bed, and the like, and use the loaded bedroom model 2-2 to perform recognition on the object in the imagephotographed from the bed room 1-2.

Configurations and operations of an electronic apparatus 10, 100according to various embodiments are described in greater detail below.

FIG. 2 is a block diagram illustrating a configuration of an electronicapparatus 100 according to an embodiment.

Referring to FIG. 2, the electronic apparatus 100 may include a sensor110, a camera 120, a storage 130, a first processor 140-1, and a secondprocessor 140-2. The electronic apparatus 100 may be a moving robotprovided with moving mechanisms or means or an auxiliary device capableof connecting and detaching to a moving device. The electronic apparatus100 may be implemented as a wearable device of various types. Inaddition, the electronic apparatus 100 may be implemented as variousterminal apparatuses such as a smartphone, a tablet personal computer(PC), a notebook PC, etc.

The sensor 110, as a configuration for determining the location of theelectronic apparatus 100, may be implemented as a light detection andranging (LiDAR) sensor, an ultrasonic sensor and the like, but is notlimited thereto.

The camera 120 is a configuration for obtaining or capturing one or moreimages on the surroundings of the electronic apparatus 100. The camera120 may be implemented as a red/green/blue (RGB) camera, a threedimensional (3D) camera, and the like.

The storage 130 is a configuration for variably storing variousinformation related to the functions of the electronic apparatus 100.The storage 130 may be implemented as a non-volatile memory such as ahard disk, a solid state drive (SSD), and a flash memory (e.g., NOR-typeflash memory or NAND-type flash memory).

The storage 130 may be stored with information 131 on the map. The mapmay refer to data indicating a physical terrain of a place in which theelectronic apparatus 100 is operated. While the map may be stored inimage form on the storage 130, it is understood that one or more otherembodiments are not limited thereto.

The information on the map, or the map itself, may include terraininformation of the place the electronic apparatus 100 is operated. Areainformation of each of the plurality of areas included in the map,additional information related to the map, and the like may also beincluded.

The terrain information may include information on a structure (e.g.,shape and/or size) of the place, information of the structure (e.g.,shape and/or size) of each of the plurality of areas included in thespace, information on a location within the place of each of theplurality of areas, and the like.

The area information may refer to information for identifying each ofthe plurality of areas. The area information may consist of or includeidentification names indicating each of the plurality of areas,identification numbers, and the like. In addition, the area informationmay include information on use of each of the plurality of areas, andeach of the plurality of areas according to area information may bedefined as, for example, living room, bath room, bedroom, and the like.

The additional information may include information on use of place(e.g., home, work, gym, etc.), location, name, user, and the like, anddata on images obtained through the camera 120 at each of the pluralityof areas.

The storage 130 may store one or more artificial intelligence models132. Specifically, a plurality of artificial intelligence models 132trained to identify objects may be stored in the storage 130 accordingto an embodiment. For example, artificial intelligence models trained toidentify objects included in the input image may be stored in plural.

The identifying an object may be understood as obtaining information onan object, such as name and type of the object. In this case,information on the objects may be information on the identified objectoutput by the plurality of artificial intelligence models thatidentified the related object.

The first processor 140-1 may be connected with the sensor 110 and thestorage 130 to control the electronic apparatus 100. Further, the firstprocessor 140-1 may be connected to the second processor 140-2 as a mainprocessor to control the second processor 140-2.

The second processor 140-2 may be connected to the camera 120, thestorage 130, and the first processor 140-1 to perform a function forobject recognition.

Referring to FIG. 2, the first processor 140-1 may determine an area inwhich the electronic apparatus 100 is located from among the pluralityof areas included in the map (operation S110). Specifically, the firstprocessor 140-1 may use information on the map divided by the pluralityof areas to identify the plurality of areas included in the map, and usesensing data obtained through the sensor 110 to determine the area inwhich the electronic apparatus 100 is located from among the pluralityof areas on the map.

The first processor 140-1 may transfer area information on thedetermined area to the second processor 140-2 connected to the firstprocessor 140-1 (operation S120). The first processor 140-1 may transferthe area information to the second processor 140-2 in the form of anelectrical signal or data.

The second processor 140-2 may, based on area information transferredfrom the first processor 140-1, load at least one artificialintelligence model of the plurality of artificial intelligence models132 (i.e., model 1, model 2 . . . ) stored in the storage 130 to thevolatile memory 145. Specifically, the second processor 140-2 may loadthe artificial intelligence model corresponding to the determined areafrom among the plurality of artificial intelligence models 132.

In this case, the second processor 140-2 may use the plurality ofartificial intelligence models 132 stored in the storage 130 and alogical mapping information between the plurality of areas to identifyand load the artificial intelligence model mapped to the determinedarea.

The logical mapping information may be information for mapping one ormore artificial intelligence models in or to each of the plurality ofareas. The logical mapping information may include information on aparameter for outputting information indicating at least one of theplurality of artificial intelligence models 132 from informationindicating each of the plurality of areas. The logical mappinginformation may include information on addresses of artificialintelligence models corresponding to each area stored in the storage130.

The logical mapping information may be pre-set by the user, and/or maybe generated and stored by the first processor 140-1 that obtained theartificial intelligence model corresponding to (mapping) each of theplurality of areas according to, for example, the embodiments of FIGS.7A to 7C, which are described below. In addition, based on the updatedresults of the artificial intelligence model corresponding to each ofthe plurality of areas according to, for example, the embodiments ofFIGS. 8A to 8C and FIGS. 9A to 9C described below, the logical mappinginformation may also be updated.

The first processor 140-1 may use the logical mapping information on thedetermined area to identify the artificial intelligence modelcorresponding to the determined area. The second processor 140-2 mayload the corresponding artificial intelligence model when (or based on)information on the identified artificial intelligence model istransferred to the second processor 140-2.

The second processor 140-2 may input the image obtained through thecamera 120 to the loaded artificial intelligence model to identify theobject (operation S140). In this case, the second processor 140-2 mayuse the output of the loaded model to identify the object included inthe image obtained through the camera 120.

Accordingly, because the second processor 140-2 may load only theartificial intelligence model corresponding to the determined area fromamong the plurality of artificial intelligence models 132 stored in thestorage 130 for use in object recognition, a relatively accurate andfast object recognition may be performed within the processingcapabilities of the second processor 140-2.

In this regard, if the area in which the electronic apparatus 100 islocated is changed, the second processor 140-2 may remove the artificialintelligence model loaded in the volatile memory 145 prior to changinglocation from the volatile memory 145, while loading a differentartificial intelligence model corresponding to the changed area toperform object recognition. That is, the second processor 140-2 may atthe very least load only the artificial intelligence model, from amongthe plurality of artificial intelligence models 132, required for eacharea in which the electronic apparatus 100 is located and use theartificial intelligence model.

The first processor 140-1 and the second processor 140-2 may beimplemented as one processor or a plurality of processors to performoperations.

Specifically, referring to FIG. 2, the processor 141 including the firstprocessor 140-1 and the second processor 140-2 may determine the area inwhich the electronic apparatus 100 is located from among the pluralityof areas on the map, load at least one model of the plurality ofartificial intelligence models 132 based on area information, and usethe loaded model to identify the object in the image obtained throughthe camera 120.

FIGS. 3A and 3B are diagrams illustrating an example of an electronicapparatus 100 identifying an area where the electronic apparatus 100 islocated from among a plurality of areas.

The first processor 140-1 may use information on the map stored in thestorage 130 and sensing data to determine the area in which theelectronic apparatus 100 is located.

As a specific example, if the sensor 110 is a LiDAR sensor, the firstprocessor 140-1 may compare sensing data received from the sensor 110and information on the map stored in the storage 130 to determine thearea in which the electronic apparatus 100 is located.

The sensing data may include information on the structure of thesurroundings of the electronic apparatus 100. Information on thesurrounding structure may include information on the structured objector the shape and/or size of things present in the surroundings.

In this case, the first processor 140-1 may compare information on thestructure (i.e., shape and/or size) of the surroundings of theelectronic apparatus 100 included in the sensing data with informationon the structure (i.e., shape and/or size) of each of the plurality ofareas on the map included in the information on the map to determine thearea in which the electronic apparatus 100 is located from among theplurality of areas on the map.

Referring to FIG. 3A, the electronic apparatus 100 implemented as arobot cleaner may use sensing data received from the sensor 110implemented as a LiDAR sensor to identify the surrounding structure 301based on the distance from the surrounding structured object or thedistance from each point (or each of a plurality of points) in theobject. Further, the electronic apparatus 100 may identify a pointcoincident with the identified surrounding structure 301 on the map 300to determine the location of the electronic apparatus 100 on the map300. The electronic apparatus 100 may use the determined location andthe location of each of the plurality of areas 300-10, 300-20, 300-30,and 300-40 on the map 300 to determine if the electronic apparatus 100is located in the kitchen 300-30 from among the plurality of areas300-10, 300-20, 300-30, and 300-40.

The first processor 140-1 may also use data on the surrounding imageobtained through the camera 120 to determine which area of the pluralityof areas the electronic apparatus 100 is located.

For example, if the information on the map includes data on a threedimensional (3D) image of each of the plurality of areas, the firstprocessor 140-1 may use the result of comparing the image of each of theplurality of areas included in the information on the map with the 3Dimage obtained through the camera 120 implemented as a 3D camera todetermine the area in which the electronic apparatus 100 is located.

Referring to FIG. 3B, the electronic apparatus 100 implemented as arobot cleaner may compare the image 302 obtained through the camera 120with the interior images of the plurality of areas 300-10, 300-20,300-30, and 300-40 stored in the storage 130 to determine if theelectronic apparatus 100 is located in the bedroom 300-20.

Alternatively, the first processor 140-1 may also identify one or moreobjects in the image obtained through the camera 120 from the area inwhich the electronic apparatus 100 is located to determine the area inwhich the electronic apparatus 100 is located.

As a specific example, the first processor 140-1 may input the imagephotographed through the camera 120 from the area in which theelectronic apparatus 100 is located to at least one of the storedplurality of artificial intelligence models 132 to identify the objectin the image. If the identified object is a sofa and a TV, the firstprocessor 140-1 may use pre-set information of one or more objects perarea to identify that the sofa and the TV correspond to the “livingroom.”

In addition, the first processor 140-1 may use the sensor 110, whichincludes an inertia sensor, an acceleration sensor, and the like, todetermine the point at which the electronic apparatus 100 is located onthe map 300, as well as determine the area including the determinedpoint from among the plurality of areas on the map as the area in whichthe electronic apparatus 100 is located.

The process of the first processor 140-1 determining the area in whichthe electronic apparatus 100 is located is not limited to theabove-described embodiments, and may be performed by various othermethods.

FIG. 4 is a diagram of a table illustrating an example of a plurality ofartificial intelligence models stored in a storage 130 of an electronicapparatus 100.

Referring to FIG. 4, in the storage 130, one or more trained artificialintelligence models, such as an air conditioner model 401 trained toidentify an air conditioner and a refrigerator model 402 trained toidentify a refrigerator, may be stored to respectively identify acorresponding object.

In the storage 130, one or more artificial intelligence models trainedto identify a plurality of objects may be stored. Referring to FIG. 4,the bedroom model 410 of the plurality of artificial intelligence modelsstored in the storage 130 may be a model (or models) capable ofidentifying objects such as an air conditioner, a TV, a bed, a chair, acup, and a glass bottle. Referring to FIG. 4, the kitchen model 430 ofthe plurality of artificial intelligence models stored in the storage130 may be may be a model (or models) capable of identifying objectssuch as an air conditioner, a refrigerator, a chair, a cup, a glassbottle, and a plate.

When the first processor 140-1 determines that the electronic apparatus100 is located in the bedroom, the second processor 140-2 may load thebedroom model 410 to the volatile memory 145. On the other hand, if thefirst processor 140-1 determines that the electronic apparatus 100 islocated in the kitchen, the second processor 140-2 may load the kitchenmodel 430 to the volatile memory 145.

Although FIG. 4 illustrates the artificial intelligence models 401, 402. . . trained to identify objects related to “home” and artificialintelligence models 410, 420, 430, 440 . . . stored to correspond toeach of the plurality of areas in the “home,” an artificial intelligencemodel for identifying one or more objects included in various places(i.e., libraries, museums, squares, playing fields, etc.) other than thehome and a plurality of artificial intelligence models stored tocorrespond to each of the plurality of areas included in the place otherthan the home may be stored in the storage 130.

The artificial intelligence model stored in the storage 130 may consistof or include a plurality of neural network layers. Each layer mayinclude a plurality of weighted values, and a calculation of the layeris performed through the calculation results of the previous layer andthe calculation of the plurality of weighted values. Examples of theneural network may include a convolutional neural network (CNN), a deepneural network (DNN), a recurrent neural network (RNN), a restrictedBoltzmann machine (RBM), a deep belief network (DBN), a bidirectionalrecurrent deep neural network (BRDNN), and a deep Q-network. Further,the neural network in the disclosure is not limited to theabove-described examples, unless otherwise specified.

The artificial intelligence model may consist of or include anontology-based data structure in which various concepts, conditions,relationships or agreed knowledge is represented in a computerprocessable form.

The artificial intelligence model stored in the storage 130 may betrained through the electronic apparatus 100 or a separate server/systemthrough various learning algorithms. The learning algorithm may be amethod that trains a predetermined target device (e.g., robot) so thatthe predetermined target device is able to determine or predict on itsown using multiple training data. Examples of the learning algorithm mayinclude supervised learning, unsupervised learning, semi-supervisedlearning, or reinforcement learning. It is understood that the learningalgorithm in the disclosure is not limited to the above-describedexamples, unless otherwise specified.

The form/type of the artificial intelligence model is not limited to theabove-described examples.

Each of the plurality of artificial intelligence models stored in thestorage 130 may include a convolutional layer and a fully-connectedlayer trained to identify at least one object based on characteristicinformation extracted through the convolutional layer.

In this case, the convolution layer may be a common layer in theplurality of artificial intelligence models 132 stored in the storage130, and the fully-connected layer may be a layer individually providedto each of the plurality of artificial intelligence models 132.

Each of the fully-connected layers consisting of or including theplurality of artificial intelligence models may be a layer trained toidentify at least one object from characteristic information output fromthe convolutional layer. In this case, each of the fully-connectedlayers may also output probability values of at least one object beingincluded in the related image for each object.

As a specific example of the electronic apparatus 100 using anartificial intelligence model including a convolutional layer and afully-connected layer to perform object recognition, the secondprocessor 140-2 may input an image obtained through the camera 120 tothe convolution layer. When characteristic information output throughthe convolution layer is then input to the fully-connected layer, thesecond processor 140-2 may use data output through the fully-connectedlayer to obtain the probability of a predetermined object being includedin the image.

The first processor 140-1 and the second processor 140-2 may, based theobtained probability being greater than a threshold value when comparingthe obtained probability with the threshold value, identify that thepredetermined object is included in the input image.

FIG. 5 is a diagram illustrating a specific example of selectivelyloading an artificial intelligence model corresponding to an identifiedarea from among a plurality of artificial intelligence models consistingof or including a convolutional layer and a fully-connected layer.

In FIG. 5, the plurality of artificial intelligence models asillustrated and described through FIG. 4 being stored in the storage 130may be assumed. Referring to FIG. 4, the plurality of artificialintelligence models stored in the storage 130 may include a bedroommodel 410, a living room model 420, a kitchen model 430, and the like.

Referring to FIG. 5, the plurality of artificial intelligence models 500stored in the storage 130 may consist of or include the convolutionallayer 501 extracting characteristic information when (or based on) dataon an image is input and the fully-connected layer 502 trained toidentify one or more objects when the extracted characteristicinformation is input.

Referring to FIG. 5, the fully-connected layer 502 may be divided into aplurality of mutually independent fully-connected layers 510, 520, 530 .. . . The mutually independent fully-connected layers 510, 520, 530 . .. may be layers trained to identify one or more objects different fromone another when (or based on) extracted characteristic information isinput.

The bedroom model 410 of FIG. 4 may include the convolutional layer 501and the fully-connected layer 510 of FIG. 5, the living room model 420of FIG. 4 may consist of the convolutional layer 501 and thefully-connected layer 520 of FIG. 5, and the kitchen model 430 of FIG. 4may consist of the convolutional layer 501 and the fully-connected layer530 of FIG. 5.

That is, the plurality of artificial intelligence models 410, 420, 430 .. . of FIG. 4 may commonly use the convolutional layer 501 while beingdivided from one another through the fully-connected layers 510, 520,530 . . . trained to identify objects different from one another.

The second processor 140-2 may load the convolutional layer and thefully-connected layer of the artificial intelligence model correspondingto the determined area to the volatile memory 145.

For example, based on determining that the electronic apparatus 100 islocated in the “bedroom,” the second processor 140-2 may load theconvolutional layer 501 and the fully-connected layer 510 to thevolatile memory 145 to use the bedroom model 410.

The plurality of artificial intelligence models 500 including a firstmodel corresponding to a first area of a plurality of areas and a secondmodel corresponding to a second area of a plurality of areas may beassumed.

The second processor 140-2 may, based on the electronic apparatus 100being located at the first area, load the convolutional layer and thefully-connected layer corresponding to the first model to the volatilememory 145, and based on the electronic apparatus 100 being located atthe second area, load the convolutional layer and the fully-connectedlayer corresponding to the second model to the volatile memory 145.

For example, based on determining that the electronic apparatus 100 islocated in the “bedroom,” the second processor 140-2 may load theconvolutional layer 501 and the fully-connected layer 510. If thelocation of the electronic apparatus 100 is determined as having changedto the “living room,” the second processor 140-2 may maintain theconvolutional layer 501 in a loaded state and load a new fully-connectedlayer 520 after removing the fully-connected layer 510 from the volatilememory 145.

The information of the map stored in the storage 130 may begenerated/updated by the first processor 140-1.

FIGS. 6A to 6D are diagrams illustrating various examples of anelectronic apparatus 100 generating information on a map.

The first processor 140-1 may obtain information on the structure of theplace the electronic apparatus 100 is located based on sensing datareceived from the sensor 110.

Referring to FIG. 6, the first processor 140-1 may control a movingmeans or mechanism of the electronic apparatus 100 for the electronicapparatus 100 to move within the related place. While the electronicapparatus 100 is in movement, the first processor 140-1 may then usesensing data received through the sensor 110, which is a LiDAR sensor,to obtain information on the structure(s) of the related place.

In this case, the first processor 140-1 may use sensing data receivedthrough the sensor 110, which may be a LiDAR sensor, to identify thedistance(s) from the surrounding structured object(s) of the electronicapparatus 100 and the distance from the points in the object(s), andobtain information on the surrounding structure (i.e., shape and/orsize) based on the identified distance.

In addition, the first processor 140-1 may obtain data on the imagephotographed through the camera 120, which may be implemented as a 3Dcamera, while the electronic apparatus 100 is in movement. The firstprocessor 140-1 may then obtain information on the structure (i.e.,shape and/or size) of the related place using the obtained image.

Referring to FIG. 6A, the first processor 140-1 may use information onthe obtained structure to obtain image data relating to a firstdimensional map 600, which appears as if a part or whole of the relatedplace is viewed from a specific direction.

The first processor 140-1 may, based on information on the obtainedstructure, divide the place in which the electronic apparatus 100 islocated to a plurality of areas.

In this case, the first processor 140-1 may use various algorithms todivide the plurality of areas of the map. For example, the firstprocessor 140-1 may identify a point where there is a dividing line or aprotrusion (or a threshold) on a floor, a point where a movable widthbecomes narrow, a point where there is a wall, a point where a wallends, a point where there is a door, and the like through sensing dataobtained through the sensor 110, which may a LiDAR sensor, and/or imagesobtained through the camera 120. The first processor 140-1 may divideeach area on the map by using the identified points as demarcationsbetween areas.

In FIG. 6B, the first processor 140-1 may determine the location of theelectronic apparatus 100 on the first dimensional map 600 through theimage (e.g., 3D image) obtained through the sensing data obtainedthrough a LiDAR sensor included in the sensor 110 and/or the camera 120.

The first processor 140-1 may then use sensing data obtained through theLiDAR sensor to identify the “point where a wall ends.” In addition, thefirst processor 140-1 may control the second processor 140-2 to inputthe image 650 (e.g., RGB image) obtained through the camera 120 to atleast one of the plurality of artificial intelligence models stored inthe storage 130 to identify “point where a wall ends” 651, “door” 652,and the like included in the image 650.

The first processor 140-1 may use the above-described points 651 and 652and the structure of the wall to divide/define one independent area600-20 including the point where the electronic apparatus 100 iscurrently located, on the first dimensional map 600. The first processor140-1 may then use other various algorithms in addition thereto torespectively divide the remaining areas 600-10, 600-30 and 600-40, onthe first dimensional map 600.

It is understood, however, that dividing the place in which theelectronic apparatus 100 is located to a plurality of areas is notlimited only to the above-described embodiments, and various othermethods and/or devices may be implemented in one or more otherembodiments.

The first processor 140-1 may generate information on the map includinginformation on the structures of each of the divided plurality of areasand store (or control to store) the information on the generated map inthe storage 130.

The first processor 140-1 may generate/store information on a mapincluding an interior image of each divided area and information on thecharacteristics of each divided area. The characteristics of eachdivided area may relate to a purpose, a size, and the like of each area.

The first processor 140-1 may add, to the information on the map, aninterior image of each of the plurality of areas obtained through thecamera 120 while the electronic apparatus 100 is positioned at each ofthe divided plurality of areas.

For example, the first processor 140 may obtain multi-angled imagesobtained through the camera 120 each time the electronic apparatus 100is located at each of the various points on the plurality of areas, andmay store the obtained multi-angled images as information on the map.

The second processor 140-2 may input the image obtained through thecamera 120 while the electronic apparatus 100 is located at each of thedivided plurality of areas to at least one of the plurality ofartificial intelligence models stored in the storage 130 to identify theobject located at each of the plurality of areas. The first processor140-1 may then obtain information on the objects identified as locatedat each of the plurality of areas as information on objects present ateach of the plurality of areas and store the obtained information in thestorage 130.

The information on objects present at each of the plurality of areas maybe information related to at least one output of the plurality ofartificial intelligence models. That is, the information on the objectsmay include the result (i.e., name, size, type, etc., of the identifiedobject) of the at least one artificial intelligence model, of theplurality of artificial intelligence models 132 stored in the storage130, identifying an object by receiving input of the images obtainedfrom the plurality of areas, and outputting information on theidentified object. The information on the objects may also includeinformation on an identity of a person if the object is a person.

As information on objects may be pre-stored for each of the plurality ofareas, the information on the objects present at each of the pluralityof areas may be stored/matched to match the information output by theplurality of artificial intelligence models to identify at least oneobject. The information on the objects present at each of the pluralityof areas may be classified and managed by categories such as name andtype (i.e., home appliance, furniture, gym equipment, etc.) forsearching/processing convenience of each of the artificial intelligencemodels.

The first processor 140-1 may use information on the objects present ateach of the plurality of areas to identify the purpose of each of theplurality of areas.

Referring to FIG. 6C, the first processor 140-1 may control the secondprocessor 140-2 to input, to the one or more artificial intelligencemodels stored in the storage 130, one or more images, obtained throughthe camera 120, of an area 600-10 while the electronic apparatus 100 islocated in the related area 600-10.

Accordingly, the first processor 140-1 and the second processor 140-2may identify that a TV 661 and a sofa 662 are located in the area600-10. The first processor 140-1 may then use the object informationfor each pre-stored area so that the one or more objects correspond toeach of the one or more areas such as “living room,” “kitchen,” and“bedroom” to identify the area corresponding to a TV 661 and a sofa 662as the “living room.” The first processor 140-1 may then identify thatthe purpose or identity of the area 600-10 is the “living room.”

The first processor 140-1 may then use objects identified in each of theremaining areas 600-20, 600-30 and 600-40 to identify that the purposeor identity of the remaining areas 600-20, 600-30 and 600-40 are the“bedroom,” the “kitchen,” and the “bathroom,” respectively.

The first processor 140-1 may divide each area or obtain information oneach area according to a user input received from the electronicapparatus 100.

For example, when an image corresponding to the first dimensional mapgenerated through sensing data of the sensor 110, which is a LiDARsensor, is displayed on a display of the electronic apparatus 100, thefirst processor 140-1 may divide the image corresponding to the firstdimensional map to a plurality of areas according to a touch input of auser for at least some zones included in the image of the displayedfirst dimensional image.

In addition, when the image corresponding to the first dimensional mapdivided into the plurality of areas is displayed on the display of theelectronic apparatus 100, the first processor 140-1 may identify thepurpose or identity of the at least one area of the plurality of areasdivided according to the touch input of the user selecting at least oneof the divided plurality of areas and the touch input of the userselecting/inputting the purpose of the selected area.

The user input for dividing each area or defining the information (e.g.,purpose) on each area may not only be directly received in theelectronic apparatus 100, but also indirectly received through anexternal apparatus such as a smartphone and a PC. In this case, theinformation on user input received through the external apparatus may bereceived by the electronic apparatus 100 from the external apparatus.

For example, FIG. 6D is a diagram illustrating an electronic apparatus100 identifying the purpose of the plurality of areas on the firstdimensional map according to a user input received through the externalapparatus 200.

Referring to FIG. 6D, the electronic apparatus 100, which is a robotcleaner, may transmit information on the first dimensional map 600divided into a plurality of areas to the external apparatus 200 afterperforming the processes as in FIGS. 6A and 6B.

Referring to FIG. 6D, the external apparatus 200 may display the firstdimensional map 600 received in a first zone 201 on a screen.

As shown in FIG. 6D, when (or based on) the touch of the user 670 on thesome zones 671 of the area 600-10 of the plurality of areas 600-10,600-20, 600-30 and 600-40 divided on the first dimensional map 600 isinput, the external apparatus 200 may identify the area 600-10 as havingbeen selected. In this case, the external apparatus 200 may visuallyindicate that the related area 600-10 has been selected by adjusting thecolor of the zone included in the selected area 600-10.

The external apparatus 200 may then display a graphical user interface(GUI) for receiving input of the purpose of the selected area 600-10 toa second zone 202 on the screen.

Referring to FIG. 6D, the GUI on the second zone 202 may include aplurality of menu items 681 that the user may select for the purpose ofthe area 600-10. In addition, the GUI on the second zone 202 may alsoinclude an item 682 for the user to directly input the purpose of thearea 600-10. When the touch of the user on the related item 682 isinput, a keyboard for the user to input a text may be displayed on thescreen of the external apparatus 200.

The external apparatus 200 may then transmit information on the selectedarea 600-10 and information on the purpose selected/input by the GUI tothe electronic apparatus 100.

The electronic apparatus 100 may then identify the purpose of theselected area 600-10 among the plurality of areas divided on the firstdimensional map 600 through the received information.

The configuration of the screen, the form (i.e., touch) of receivinguser input, and the like illustrated and described through FIG. 6D aremerely an example, and various technical methods that are generallyknown may be applicable in addition thereto. Further, although FIG. 6Dillustrates receiving a user input when the first dimensional map 600has already been divided into a plurality of areas, receiving a userinput to divide the first dimensional map 600 to a plurality of areasthrough the external apparatus may also be possible in variousembodiments.

The electronic apparatus may newly define/obtain the plurality ofartificial intelligence models corresponding to each of the plurality ofareas divided through the process of FIG. 6B and/or FIG. 6C. To thisend, the electronic apparatus may use the artificial intelligence modeltrained to identify the plurality of objects that is pre-stored in thestorage.

Specifically, when an artificial intelligence model trained to identifya plurality of objects is stored in the storage 130, the secondprocessor 140-2 may input an image obtained through the camera 120 whilethe electronic apparatus 100 is located at each of the plurality ofareas to the artificial intelligence model to identify objects presentat each of the plurality of areas. Although, the process may beperformed separate from the process of FIG. 6C where an object isidentified in the place the electronic apparatus is located, the processmay also be performed together with the process of FIG. 6C.

Further, the second processor 140-2 may transfer (or provide)information on the identified object to the first processor 140-1. Inthis case, the first processor 140-1 may, based on information on theidentified object transferred from the second processor 140-2, obtainthe artificial intelligence model corresponding to each of the pluralityof areas from the stored artificial intelligence models.

FIGS. 7A to 7C are diagrams illustrating an example of an electronicapparatus 100 obtaining an artificial intelligence model correspondingto each of a plurality of areas using an object identified in each ofthe plurality of areas.

FIG. 7A is a diagram of a table briefly illustrating information on theobjects identified at each of the plurality of areas being stored in thestorage 130 as information on objects present at each of the pluralityof areas.

Referring to FIG. 7A, an air conditioner 11, a TV 13, a bed 21, a chair23 and the like are present in the bedroom 51, and an air conditioner11, a TV 13, a sofa 22, and the like are present in the living room 52.

FIG. 7B is a diagram illustrating a data structure of an artificialintelligence model 700 stored in the storage 130 prior to obtaining theplurality of artificial intelligence models corresponding to each of theplurality of areas.

Referring to FIG. 7B, the artificial intelligence model 700 may becomposed as a fully-connected layer 702 for identifying a plurality ofobjects using a convolutional layer 701 and characteristic informationextracted from the convolutional layer 701.

FIG. 7B is a diagram illustrating a node 711 outputting the probabilityof an air conditioner being included in the input image, a node 712outputting the probability of a refrigerator being included in the inputimage, a node 713 outputting the probability of a TV being included inthe input image, a node 721 outputting the probability of a bed beingincluded in the input image, a node 722 outputting the probability of asofa being included in the input image, and a node 723 outputting theprobability of a chair being included in the input image. Additionally,node 731, node 732, and node 733 are nodes respectively related to acup, a glass bottle, and a plate.

The first processor 140-1 may, based on information on the objectspresent at a first area from information on objects present at each ofthe plurality of areas stored in the storage 130, identify that a firstobject is present at the first area.

In this case, the first processor 140-1 may identify a part of thefully-connected layer trained to identify the first object of thefully-connected layer of the artificial intelligence model stored in thestorage 130.

The first processor 140-1 may then obtain (define) a first modelincluding a part of the identified fully-connected layer and theconvolutional layer of the artificial intelligence model stored in thestorage 130 and store the first model in the storage 130. In this case,the first processor 140-1 may generate logical mapping informationconnecting (matching) the first model with the first area and store thegenerated information in the storage 130.

The first processor 140-1 may, based on information on an object presentin a second area from information on objects present at each of theplurality of areas stored in the storage 130, identify that a secondobject is present at the second area.

The first processor 140-1, in this case, may identify a different partof the fully-connected layer trained to identify the second object fromthe fully-connected layer of the artificial intelligence model stored inthe storage 130.

The first processor 140-1 may then obtain (define) a second modelincluding a different part of the identified fully-connected layer andthe convolutional layer of the artificial intelligence model stored inthe storage 130 and store the second model in the storage 130. In thiscase, the first processor 140-1 may generate logical mapping informationconnecting (matching) the second model with the second area and storethe generated information in the storage 130.

For example, the first processor 140-1 may use information in FIG. 7A toidentify that an air conditioner 11, a TV 13, a bed 21, a chair 23, andthe like are present in the “bedroom” 51.

The first processor 140-1 may define a new fully-connected layer 751′including the nodes 711, 713, 721 and 723 related to the air conditioner11, the TV 13, the bed 21, and the chair 23 and the part used in aninference process to generate an output of the corresponding nodes 711,713, 721 and 723 from the fully-connected layer 702 illustrated in FIG.7B.

Although the fully-connected layer 751′ may be related to the inferenceprocess for generating output of the node 712 related to therefrigerator of the fully-connected layer 702, the fully-connected layer751′ may not include parts unrelated to the inference process forgenerating output of nodes 711 and 713 related to the air conditioner 11and the TV 13.

Further, referring to FIG. 7C, the first processor 140-1 may obtain anew bedroom model 751 including the convolutional layer 701 and thefully-connected layer 751′.

For example, the first processor 140-1 may use information of FIG. 7A toidentify that an air conditioner 11, a TV, 13, a sofa 22, and the likeare present in the “living room” 52.

The first processor 1140-1 may define/obtain a new fully-connected layer752′ including the nodes 711, 713 and 722 related to the air conditioner11, the TV 13, the sofa 22 and the part used in the inference process togenerate an output of the corresponding nodes 711, 713, and 722 from thefully-connected layer 702 illustrated in FIG. 7B.

Although the fully-connected layer 752′ may be related to the inferenceprocess for generating output of the node 721 related to the bed 21 ofthe fully-connected layer 702, the fully-connected layer 752′ may notinclude parts unrelated to the inference process for generating outputof the node 722 related to the sofa 22.

Referring to FIG. 7C, the first processor 140-1 may obtain a living roommodel 752 including the convolutional layer 701 and the fully-connectedlayer 752′.

The first processor 140-1 may then store the obtained artificialintelligence model in the storage 130 among artificial intelligencemodels corresponding to each area.

The first processor 140-1 may store the logical mapping information thatmaps the fully-connected layer 751′ included in the bedroom model 751 ofthe fully-connected layer 702 with the bedroom 51 in the storage 130. Inaddition, the first processor 140-1 may store the logical mappinginformation that maps the fully-connected layer 752′ included in theliving room model 752 of the fully-connected layer 702 with the livingroom 52 in the storage 130.

If, for example, the electronic apparatus 100 is determined as beinglocated in the bedroom 51, the second processor 140-2 may then load onlythe bedroom model 751 of the artificial intelligence models 700 storedin the storage 130 to the volatile memory 145 according to the controlof the first processor 140-1. Specifically, the second processor 140-2may load the fully-connected layer 751′ that is mapped with the logicalmapping information related to the bedroom 51 together with theconvolutional layer 701.

Alternatively, if, for example, the electronic apparatus 100 isdetermined as being located in the living room 52, the second processor140-2 may load only the living room model 752 of the artificialintelligence models 700 stored in the storage 130 to the volatile memory145 according to the control of the first processor 140-1. Specifically,the second processor 140-2 may load the fully-connected layer 752′ thatis mapped with the logical mapping information related to the livingroom 52 together with the convolutional layer 701.

In addition to the process of FIG. 6C of identifying an object presentin the place in which the electronic apparatus 100 is located whilegenerating information on the map, the first processor 140-1 may performscanning identifying objects present at each of the plurality of areas.In this case, the first processor 140-1 may control a moving means ormechanism (e.g., wheels) of the electronic apparatus 100 to move aroundon the plurality of areas while controlling the second processor 140-2to identify objects present at each of the plurality of areas from theimage obtained through the camera 120 at each of the plurality of areas.

The first processor 140-1 may either perform scanning according toreceived user command, or may perform the above-described scanningaccording to a pre-set interval regardless of user command.

In addition, the first processor 140-1 may only perform theabove-described scanning when there is no user in the place includingthe plurality of areas. In this case, the first processor 140-1 mayidentify that there is no user at the corresponding place through thereceived user input. In addition, the first processor 140-1 may controlthe second processor 140-2 to recognize whether a user is presentthrough an image obtained through the camera 120 at the plurality ofareas, and identify that no user is present at the corresponding placebased on the output of the artificial intelligence model loaded by thesecond processor 140-2. In this case, the artificial intelligence modelmay be an artificial intelligence model trained to identify whether auser is included in the input image.

The electronic apparatus 100 may update the artificial intelligencemodel corresponding to each of the plurality of areas according to thescanning results on the plurality of areas.

Specifically, the second processor 140-2 may input the image obtainedthrough the camera 120 while the electronic apparatus 100 is located atone area of the plurality of areas to at least one of the plurality ofartificial intelligence models loaded in the volatile memory 145 toidentify the object present in the corresponding area, and may transferinformation on the identified object to the first processor. In thiscase, the plurality of artificial intelligence models may not be loadedsimultaneously in the volatile memory 145, but may be loadedsequentially by one or two models at a time.

The first processor 140-1 may then update the artificial intelligencemodel corresponding to the related area based on information of theidentified object transferred from the second processor 140-2.

For example, when information on the objects present at each of theplurality of areas is stored, the first processor 140-1 may, based oninformation on the objects present at each of the plurality of areasstored in the storage 130, determine at least one object present at theone area of the plurality of areas, and based on information on theidentified object from the corresponding area transferred from thesecond processor 140-2, determine the unidentified object in thecorresponding area from the determined objects.

The first processor 140-1 may then remove the trained part to identifythe unidentified object from the artificial intelligence modelscorresponding to the related area of the plurality of artificialintelligence models to update the artificial intelligence modelcorresponding to the related area.

FIGS. 8A to 8C are diagrams illustrating an example of an electronicapparatus 100 updating an artificial intelligence model corresponding toa related area in case an object present in one area of a plurality ofareas is identified as not being present at the related area any longer.

FIGS. 8A to 8C illustrate an example of information on objects presentat each of the plurality of areas being stored in the storage 130 as inFIG. 7A. The information on the objects present at each of the pluralityof areas stored in the storage 130 may be pre-set information or may beinformation on the objects identified at each of the plurality of areasthrough the process of FIG. 6C. In addition, FIGS. 8A to 8C illustratean example of the plurality of artificial intelligence modelscorresponding to each of the plurality of areas being composed as inFIG. 7C and stored in the storage 130.

Referring to FIG. 8A, the first processor 140-1 of the electronicapparatus 100 implemented as a robot cleaner may control a moving meansor mechanism (device) of the electronic apparatus 100 so that theelectronic apparatus 100 may move around a plurality of areas 800-10,800-20, 800-30 and 800-40 on the place indicated by a map 800.

Further, while the electronic apparatus 100 is located at each of theplurality of areas 800-10, 800-20, 800-30 and 800-40, the secondprocessor 140-2 may load at least one of the plurality of artificialintelligence models stored in the storage 130, and input an imageobtained through the camera 120 to the loaded artificial intelligencemodel to identify the object located at each of the plurality of areas.

To this end, the first processor 140-1 may control the movement of theelectronic apparatus 100 to pass all of the plurality of areas 800-10,800-20, 800-30 and 800-40 at least one or more times.

Referring to FIG. 8 the electronic apparatus 100 may identify an “airconditioner” 11 and a “TV” 13 in the “living room” 800-10. However,referring to FIG. 8A, because the “sofa” 22 that previously was presentin the “living room” 800-10 is no longer present in the “living room”800-10, the electronic apparatus 100 may no longer be able to identifythe “sofa” 22 in the “living room” 800-10.

When the “sofa” 22 is no longer identified in the “living room” 800-10,for example, when the “sofa” 22 is not identified in the “living room”800-10 for a threshold time (e.g., threshold time may be variouslypreset to two days, one week, etc.), the first processor 140-1 mayupdate information on the objects present in the “living room” 52 fromthe information on the objects present at each of the plurality of areasstored in the storage 130.

Accordingly, referring to FIG. 8B, information on the objects present inthe “living room” 52 stored in the storage 130 may be updated so as tonot include the “sofa” 22 anymore.

In this case, referring to FIG. 8C, the first processor 140-1 may obtainan artificial intelligence model 852 with the part trained to identifythe “sofa” 22 of the fully-connected layer 702 removed or obtained fromthe artificial intelligence model 752 corresponding to “living room” 52.

The part trained to identify the “sofa” 22 may refer to the part used inthe inference process to generate an output of the node 722 on the“sofa” 22 of the fully-connected layer 702. However, although the partmay be used in the inference process to generate an output of the node722, the first processor 140-1 may not remove the related part if thepart is also used in the inference process to generate an output ofnodes related to an “air conditioner” 11 and a “TV” 13.

The first processor 140-1 may then update (or remove) an artificialintelligence model 852 that is obtained from the artificial intelligencemodel 752 corresponding to the “living room” 52 to store in the storage130.

When (or based on) information on objects present at each of theplurality of areas is stored in the storage 130, the first processor140-1 may, based on information on the objects present at each of theplurality of areas stored in the storage 130, determine at least oneobject present at one area of the plurality of areas. Further, based oninformation on the identified object from the related area transferredfrom the second processor 140-2, the first processor 140-1 may determinethe object not included among the determined at least one object of theidentified objects from the related area.

In this case, the first processor 140-1 may add a fully-connected layertrained to identify the object not included among the determined atleast one object to the artificial intelligence model corresponding tothe related area from among the plurality of artificial intelligencemodels to update the artificial intelligence model corresponding to therelated area.

FIGS. 9A to 9C are diagrams illustrating an example of updating anartificial intelligence model corresponding to a related area in case anew object is identified as being added to one area of a plurality ofareas.

The embodiment of FIGS. 9A to 9C describe a situation of information onthe objects present in the “living room” 52 as in the right side tableof FIG. 8B being stored in the storage 130. In addition, an artificialintelligence model corresponding to the “living room” 52 being stored asa living room model 852 of FIG. 8C may be assumed.

Referring to FIG. 9A, the electronic apparatus 100 may identify the “airconditioner” 11 and the “TV” 13 in the “living room” 800-10. Inaddition, referring to FIG. 9A, because the “chair” 23 that was notpresent in the previous “living room” 800-10 is now present in the“living room” 800-10, the electronic apparatus 100 may also identify the“chair” 23 in the “living room” 800-10.

When the “chair” 23 that was not present in the previous “living room”800-10 is newly identified, the first processor 140-1 may updateinformation on the objects present in the “living room” 52 from theinformation on objects present at each of the plurality of areas storedin the storage 130.

Accordingly, referring to FIG. 9B, the information on the objectspresent in the “living room” 52 stored in the storage 130 may be updatedto include the “chair” 23 in addition to the “air conditioner” 11 andthe “TV” 12.

In this case, referring to FIG. 9C, the first processor 140-1 may addthe trained part (e.g., including the node 723) to identify the “chair”23 of the fully-connected layer 702 to the artificial intelligence model852 corresponding to the “living room” 52 to obtain an artificialintelligence model 952. The part trained to identify the “char” 23 mayrefer to the part used in the inference process to generate an output ofthe node 723 related to the “chair” 23 of the fully-connected layer 702.

Although all of nodes of the “bed” 21, the “chair” 23, and the “sofa” 22are illustrated as comprised in the one independent fully-connectedlayer in FIG. 7B, the node for identifying the “chair” may be includedin a separate independent fully-connected layer according to anotherembodiment. In this case, the part trained to identify the “chair” 23may be an independent fully-connected layer including the node toidentify the “chair” 23.

Further, the first processor 140-1 may update the artificialintelligence model corresponding to the “living room” 52 to an obtainedartificial intelligence model 952 and store the same in the storage 130.

When the artificial intelligence or electronic apparatus 100 is locatedat one area of the plurality of areas, if (or based on) an object of therelated area is not identified even when the second processor 140-2loads the artificial intelligence model corresponding to the relatedarea to the volatile memory 145, the second processor 140-2 maysequentially load a different artificial intelligence model of theplurality of artificial intelligence models to identify the relatedobject.

If (or based on) the related object is identified through a differentartificial intelligence model, the first processor 140-1 may use theinformation on the identified object to change the information on theobjects present at the related area from information on objects presentat each of the plurality of areas. In addition, the first processor140-1 may use information on the changed object to update the artificialintelligence model corresponding to the related area. As a specificexample, the fully-connected layer trained to recognize the identifiedobject may be added to the artificial intelligence model correspondingto the related area.

The information on the objects present at each of the plurality of areasmay be generated and updated by user input received by the electronicapparatus 100 and/or data received by the electronic apparatus 100 fromthe external apparatus and stored in the storage 130. In this case, thefirst processor 140-1 may also use the generated/updated “information onobjects present at each of the plurality of areas” to update theartificial intelligence model corresponding to each of the plurality ofareas.

The obtaining and/or updating of the artificial intelligence modelscorresponding to each of the plurality of areas may be performed basedon changes in information on only “fixed type objects” (and notnon-fixed type objects) present at each of the plurality of areas,according to one or more embodiments.

The fixed type object may refer to objects with nearly no movement in anactual life of a person such as, for example, a bed, a sofa, a TV, andthe like, while a non-fixed type object may refer to objects withfrequent movement in an actual life of a person such as, for example, acup, a plate, a ball, a toy, and the like.

The second processor 140-2 may, even if the electronic apparatus 100 ispositioned at any area of the plurality of areas, always load theartificial intelligence model trained to identify the non-fixed typeobject to the volatile memory 145. In this case, the plurality ofartificial intelligence models stored in the storage 130 correspondingto each of the plurality of areas may be artificial intelligence modelstrained to identify the fixed type objects.

FIG. 10 is a block diagram illustrating a detailed configuration of anelectronic apparatus 100 including a first processor 140-1 and a secondprocessor 140-2 according to an embodiment.

Referring to FIG. 10, the electronic apparatus 100 may further includeat least one of a first memory 150-1, a second memory 150-2, acommunicator including circuitry 160, a user inputter 170 (or user inputdevice), an outputter 180 (or output device), and a driving controller190 in addition to the sensor 110, the camera 120, the storage 130, thefirst processor 140-1, and the second processor 140-2.

The sensor 110 may be implemented as a light detection and ranging(LiDAR) sensor, an ultrasonic sensor, and the like. When the sensor 110is implemented as a LiDAR sensor, the sensing data generated accordingto the sensed result of the sensor 110 may include information on thestructured objects present in the surroundings and/or the distancebetween at least a part of things (or objects) and the electronicapparatus 100. Information on the above-described distance may form orbe the basis of information on structure (i.e., shape and/or size) ofstructured objects/things present in the surroundings of the electronicapparatus 100.

The camera 120 may be implemented as a RGB camera, a 3D camera, and thelike. The 3D camera may be implemented as a time of flight (TOF) cameraincluding a TOF sensor and an infrared (IR) light. The 3D camera mayinclude an IR stereo sensor. The camera 120 may include sensors such asa charge-coupled device (CCD) and a complementary metal-oxidesemiconductor (CMOS), but is not limited thereto. If the camera 120includes a CCD, the CCD may be implemented as a red/green/blue (RGB)CCD, an IR CCD, and the like.

The information on the map stored in the storage 130 may includeinformation on the purpose of each of the plurality of areas. Thepurpose of each of the plurality of areas may relate to the “livingroom,” the “bedroom,” the “kitchen,” the “bathroom,” and the like if themap including the plurality of areas relates to, for example, a map ofthe “home.”

In the storage 130, information on objects present in each of theplurality of areas on the map may be stored in addition to the pluralityof artificial intelligence models and information on the map.

Information on the objects present at each of the plurality of areas mayinclude the name, type and/or the like of the object present at each ofthe plurality of areas. The information on the objects may includeinformation on the identity of a person if the object is a person. Theinformation on the objects present at each of the plurality of areas maybe stored/managed to match information output by the plurality ofartificial intelligence models to identify at least one object from theimages obtained through the camera 120 in the plurality of areas.

The information on the objects present at each of the plurality of areasmay be pre-stored or obtained by the electronic apparatus 100 performingobject recognition at each of the plurality of areas.

The first processor 140-1 may consist of or include one or a pluralityof processors. The one or plurality of processors may be a genericprocessor such as a central processing unit (CPU) and an applicationprocessor (AP), and a graphics dedicated processor such as a graphicsprocessing unit (GPU) and a vision processing unit (VPU), and the like.

The first processor 140-1 may control various configurations included inthe electronic apparatus 100 by executing at least one instructionstored in the first memory 150-1 or the storage 130 connected to thefirst processor 140-1.

To this end, information or instruction to control the variousconfigurations included in the electronic apparatus 100 may be stored inthe first memory 150-1.

The first memory 150-1 may include a read-only memory (ROM), a randomaccess memory (RAM), e.g., dynamic RAM (DRAM), synchronous DRAM (SDRAM),and double data rate SDRAM (DDR SDRAM)) and the like, and may beimplemented together with the first processor 140-1 on one chip 1001.

The second processor 140-2 may also be implemented as one or moreprocessors. The second processor 140-2 may be implemented as anartificial intelligence dedicated processor such as a neural processingunit (NPU), and may include a volatile memory 145 for loading at leastone artificial intelligence model. The volatile memory 145 may beimplemented as one or more status RAM (SRAM).

The second memory 150-2 may be stored with information or instructionsfor controlling the function performed by the second processor 140-2 forobject recognition. The second memory 150-2 may also include a ROM, aRAM (e.g., DRAM, SDRAM, DDR SDRAM) and the like, and may be implementedtogether with the second processor 140-2 on one chip 1002.

The communicator including circuitry 160 is a configuration for theelectronic apparatus 100 to send and receive signals/data by performingcommunication with at least one external apparatus.

The communicator including circuitry 160 may include a wirelesscommunication module, a wired communication module, and the like.

The wireless communication module may include at least one of a Wi-Ficommunication module, a Bluetooth module, an infrared data association(IrDA) communication module, a 3^(rd) generation (3G) mobilecommunication module, a 4^(th) generation (4G) mobile communicationmodule, and a 4G long term evolution (LTE) communication module toreceive content from an external server or external apparatus.

The wired communication module may be implemented as wired ports suchas, for example, a thunderbolt port, a USB port, and the like.

The first processor 140-1 may use data received externally through thecommunicator including circuitry 160 to generate/update information onthe map.

The first processor 140-1 and/or the second processor 140-2 may use datareceived externally through the communicator including circuitry 160 togenerate/update the artificial intelligence model corresponding to eachof the plurality of areas.

The first processor 140-1 may, based on receiving a control signalthrough the communicator including circuitry 160, control the secondprocessor 140-2 to start/end recognition of an object located in atleast one area of the plurality of areas. At this time, the controlsignal may have been received from a remote control for controlling theelectronic apparatus 100 or a smartphone stored with remote controlapplications on the electronic apparatus 100.

The at least part of the plurality of artificial intelligence modelsstored in the storage 13 may be artificial intelligence models includedin the data received from the external apparatus such as a serverapparatus to the electronic apparatus 100 through the communicatorincluding circuitry 160.

When the second processor 140-2 is not able to recognize the objectincluded in the image obtained through the camera 120 despite using allof the plurality of artificial intelligence models stored in the storage130, the first processor 140-1 may transmit data on the obtained imageto the server apparatus through the communicator including circuitry160.

At this time, the data on the results of recognizing the object includedin the obtained image may be received by the electronic apparatus 100from the server apparatus through the communicator including circuitry160.

Further, the first processor 140-1 may receive data of the artificialintelligence model trained to identify recognized objects from theexternal apparatus through the communicator including circuitry 160, andmay store the received artificial intelligence model in the storage 130.

When data indicating the location of the electronic apparatus 100 isreceived from the external apparatus through the communicator includingcircuitry 160, the first processor 140-1 may use the received data todetermine in which area the electronica apparatus 100 is located.

Based on a user input received through the user inputter 170, the firstprocessor 140-1 may update information on the map and/or at least a partof the information on the objects present in at least one of theplurality of areas. In addition, the processor 140-1 may use datareceived according to user input to generate information on the map.

Based on the user input received through the user inputter 170, thefirst processor 140-1 may control the moving means or mechanism of theelectronic apparatus 100 to move around at least one area of theplurality of areas, and may control the second processor 140-2 tostart/end object recognition.

When a user input indicating the location of the electronic apparatus100 is received through the user inputter 170, the first processor 140-1may use the received user input to determine in which area theelectronic apparatus 100 is located.

The user inputter 170 may include one or more of a button, a keyboard, amouse, and the like. In addition, the user inputter 170 may include atouch panel implemented together with a display or a separate touchpanel.

The user inputter 170 may include a microphone to receive a voice inputof a user command or information, and may be implemented together withthe camera 120 to recognize a user command or information in motion orgesture form.

The outputter 180 may be a configuration for the electronic apparatus100 to provide the obtained information to the user.

For example, the outputter 180 may include a display, a speaker, anaudio terminal, and the like to visually/audibly provide the user withobject recognition results.

The driving controller 190, as a configuration to control the movingmeans or mechanism of the electronic apparatus 100, may include anactuator for providing power to the moving means or mechanism of theelectronic apparatus 100. The first processor 140-1 may control themoving means or mechanism of the electronic apparatus 100 through thedriving controller 190 to move the electronic apparatus 100.

The electronic apparatus 100 may further include various configurationsnot illustrated in FIG. 10 in addition thereto.

The above-described embodiments have been described based on theplurality of artificial intelligence models being stored in the storage130, but it is understood that one or more other embodiments are notlimited thereto. For example, according to one or more otherembodiments, the plurality of artificial intelligence models may bestored in an external server apparatus and object recognition for eacharea may be possible through the electronic apparatus 100.

In addition, the electronic apparatus may perform object recognition foreach area through communication with an external terminal apparatusimplemented as a smartphone, tablet PC, and the like.

FIG. 11 is a diagram illustrating various embodiments of an electronicapparatus 100 performing object recognition based on communication withexternal apparatuses including a server apparatus 300 and an externalterminal apparatus 200-1 and 200-2.

Referring to FIG. 11, the electronic apparatus 100, which is a robotcleaner in the present example, may perform communication with externalapparatuses 200-1 and 200-2 such as a smartphone and a server apparatus300. In this case, the electronic apparatus 100 may also performcommunication with external apparatuses 200-1, 200-2 and 300 through arelay apparatus 400 configured with routers and the like.

The electronic apparatus 100 may perform recognition of objects presentat a plurality of areas based on the external apparatus 200-1, which isa smartphone, or a control signal received from the external apparatus200-2. In addition, the electronic apparatus 100 may transmitinformation on the recognized object to the external apparatus 200-1and/or the external apparatus 200-2.

FIG. 11 illustrates each of the plurality of artificial intelligencemodels trained to recognize at least one object being stored in a serverapparatus 300 and not (or not necessarily) the storage 130 of theelectronic apparatus 100.

In this case, information on the plurality of artificial intelligencemodels (i.e., information on objects recognizable by each of theplurality of artificial intelligence models) stored in the serverapparatus 300 may be received by the electronic apparatus 100 from theserver apparatus 300 through the communicator including circuitry 160.

The processor 140-1 may then select the artificial intelligence modelcorresponding to the area determined as where the electronic apparatus100 is located from the plurality of artificial intelligence modelsstored in the server apparatus 300.

The first processor 140-1 may then control the communicator includingcircuitry 160 to transmit information on the selected artificialintelligence model to the server apparatus 300.

When data on the selected artificial intelligence model is received fromthe server apparatus 300 through the communicator including circuitry160, the first processor 140-1 may control the second processor 140-2 toload the selected artificial intelligence model (data) to the volatilememory 145. The second processor 140-2 may then perform objectrecognition by inputting the image obtained through the camera 120 tothe loaded artificial intelligence model.

In this case, the first processor 140-1 may store the data on thereceived artificial intelligence model to the storage 130.

According to another embodiment, the electronic apparatus 100 mayinclude one processor.

FIGS. 12A and 12B are block diagrams illustrating a configuration of anelectronic apparatus 100 including a processor 140′.

Referring to FIG. 12A, the electronic apparatus 100 may include aprocessor 140′ controlling the electronic apparatus 100 and connected toa sensor 110′, a camera 140′ and a storage 130′. Additionally, theelectronic apparatus 100 may include the sensor 110′, the camera 140′and the storage 130′.

The processor 140′ may be implemented as a generic processor such as aCPU and an AP, a graphics dedicated processor such as a GPU and a visionprocessing unit (VPU), an artificial intelligence dedicated processorsuch as an NPU, or the like, and may include a volatile memory forloading at least one artificial intelligence model.

The processor 140′ may perform operations performed by the firstprocessor 140-1 or the second processor 140-2 as in the variousembodiments described above.

Specifically, the processor 140′ may identify a plurality of areasincluded in the map based on information on the map stored in a storage130′, determine an area in which the electronic apparatus 100 is locatedfrom the plurality of areas based on sensing data received from thesensor 110′, and load the artificial intelligence model corresponding tothe determined area from the plurality of artificial intelligence modelsstored in the storage 130 to the volatile memory. The processor 140′ mayinput the image obtained through the camera 120′ to the loadedartificial intelligence model to identify the object.

Referring to FIG. 12B, the electronic apparatus 100 including theprocessor 140′ may further include a memory 150′, a communicator 160′including circuitry, a user inputter 170′, an outputter 180′, a drivingcontroller 190′, and the like connected to the processor 140′ as in FIG.10.

The memory 150′ is a configuration for storing an operating system (OS)for controlling the overall operations of elements of the electronicapparatus 100 and data related to the elements of the electronicapparatus 100. The memory 150′ may include at least one instructionrelated to the one or more elements of the electronic apparatus 100.

The memory 150′ may include a ROM, a RAM (e.g., DRAM, SDRAM, and DDRSDRAM) and the like, and may be implemented to connect with theprocessor 140′ in one chip 1201.

FIGS. 13 to 15 describe a control method of an electronic apparatus 100according to one or more embodiments.

FIG. 13 is a flowchart illustrating a control method of an electronicapparatus 100 using an object recognition model according to anembodiment of the disclosure.

Referring to FIG. 13, the control method may identify a plurality ofareas included in the map based on information on the map stored in thestorage of the electronic apparatus 100 (operation S1310). Theinformation on the map may include at least one of information on thestructure of the place, information on the structure of each of theplurality of areas included in the map, information on the location onthe map of each of the plurality of areas, information on the purpose ofeach of the plurality of areas, and the like.

Then, the area in which the electronic apparatus 100 is located fromamong the plurality of areas may be determined based on the sensing datareceived from the sensor (operation S1320).

In this case, the area in which the electronic apparatus 100 is locatedmay be determined using information on the map stored in the storage andsensing data received through the sensor. As a specific example, if thesensor is a LiDAR sensor, the sensing data received from the sensor andthe information on the map stored in the storage may be compared and thearea in which the electronic apparatus 100 is located may be determined.

At this time, information on the structure (i.e., shape and/or size) ofthe surroundings of the electronic apparatus 100 included in the sensingdata may be compared with information on the structure (i.e., shapeand/or size) of each of the plurality of areas on the map included inthe information on the map, and the area in which the electronicapparatus 100 is located from the plurality of areas on the map may bedetermined.

By using data on the surrounding image obtained through the camera, itmay be possible to determine in which area of the plurality of areas theelectronic apparatus 100 is located. For example, if information on themap includes data on the 3D image of the plurality of areas, the area inwhich the electronic apparatus 100 is located may be determined usingthe results of comparing images of each of the plurality of areasincluded in the information on the map with the 3D image obtainedthrough the camera, implemented as a 3D camera.

In addition, the area in which the electronic apparatus 100 is locatedmay be determined by identifying one or more objects from the imageobtained through the camera in areas the electronic apparatus islocated.

As a specific example, the object in the image may be identified byinputting the image photographed through the camera in the area in whichthe electronic apparatus 100 is located to at least one of the storedplurality of artificial intelligence models. If the identified object isa bed, one or more objects per area may use pre-stored information toidentify that the bed corresponds with the “bedroom.” Further, the areain which the electronic apparatus 100 is located may be determined asthe “bedroom.” The area in which the electronic apparatus 100 is locatedmay then be determined as the “bedroom.”

In addition, a point where the electronic apparatus 100 is located maybe determined on the map using an inertia sensor, an accelerationsensor, and the like, and the area including the determined point, fromamong the plurality of areas on the map, may be determined as an area inwhich the electronic apparatus 100 is located.

The process of determining an area in which the electronic apparatus 100is located is not limited to the above-described embodiments, and othervarious methods may be applicable.

The control method may include loading the artificial intelligence modelcorresponding to the determined area from the plurality of artificialintelligence models stored in the storage to the volatile memory(operation S1330).

Each of the plurality of artificial intelligence models may include aconvolutional layer and a fully-connected layer trained to identify anobject based on characteristic information extracted from theconvolutional layer. The convolutional layer, at this time, may be acommon layer of the plurality of artificial intelligence models, and thefully-connected layer may be a layer provided individually to each ofthe plurality of artificial intelligence models.

If (or based on) the electronic apparatus 100 is located at a firstarea, the convolutional layer and the fully-connected layercorresponding to a first model from among the plurality of artificialintelligence models may be loaded to the volatile memory. If (or basedon) the electronic apparatus 100 is located at a second area, theconvolutional layer and the fully-connected layer corresponding to asecond model from among the plurality of artificial intelligence modelsmay be loaded to the volatile memory.

The first model may correspond to the first area of the plurality ofareas, and the second model may correspond to the second area of theplurality of areas. In this case, the logical mapping information thatmaps the first model in the first area and the logical mappinginformation that maps the second model in the second area may be storedin the storage, and the control method may use the logical mappinginformation stored in the storage to load the artificial intelligencemodel corresponding to each area.

The image obtained through the camera may then be input to the loadedartificial intelligence model to identify the object (operation S1340).Specifically, because information on the objects output by the loadedartificial intelligence model may be obtained, the information on theobjects may be varied according to the artificial intelligence modelproperties such as name, type, and the like of the object.

The control method may generate information on the map of the place inwhich the electronic apparatus 100 is located to store in the storage.In addition, the control method may newly obtain/define the artificialintelligence model corresponding to each of the plurality of areasincluded in the map.

FIG. 14 is a flowchart illustrating an embodiment of a control method ofan electronic apparatus 100 according to an embodiment generatinginformation on a map, and identifying objects present in each of aplurality of areas to obtain an artificial intelligence modelcorresponding to each of the plurality of areas.

Referring to FIG. 14, the control method may obtain information on thestructure of the place in which the electronic apparatus 100 is locatedbased on sensing data received from the sensor (operation S1410).

In an example, information on the structure (i.e., shape and/or size) ofthe place in which the electronic apparatus is located may be obtainedusing sensing data received from the sensor, implemented as a LiDARsensor.

Then, based on information on the obtained structure, the place in whichthe electronic apparatus 100 is located may be divided into a pluralityof areas (operation S1420).

In this case, the plurality of areas may be divided on the map using afirst algorithm. For example, a point where there is a dividing line ora protrusion (or threshold) on a floor, a point where a movable widthbecomes narrow, a point where there is a wall, a point where a wallends, a point where there is be a door, and the like may be identifiedthrough sensing data obtained through the sensor, which may be a LiDARsensor, and/or an image obtained through a camera. Further, using theidentified points as demarcations between areas, each area on the mapmay be divided. However, other methods may be applied in additionthereto.

Then, information on the map including information on the structure ofeach of the divided plurality of areas may be generated, and theinformation on the generated map may be stored in the storage (operationS1430).

The control method may, when artificial intelligence models trained toidentify a plurality of objects are stored in the storage of theelectronic apparatus 100, input the image obtained through the camerawhile the electronic apparatus 100 is located in each of the pluralityof areas to the stored artificial intelligence model and identify theobject present at each of the plurality of areas (operation S1440).

The stored artificial intelligence model may include the convolutionallayer and the fully-connected layer trained to identify a plurality ofobjects based on characteristic information extracted from theconvolutional layer.

Then, a first object may be identified in the first area of theplurality of areas, and a second object may be identified in the secondarea of the plurality of areas.

Then, the artificial intelligence model corresponding to each of theplurality of areas may be obtained from the stored artificialintelligence model based on information on the identified object(operation S1450).

Specifically, when (or based on) a first object of a plurality ofobjects is identified as present in the first area based on informationon the objects present at the first area of the plurality of areas, thefirst model including the part trained to identify the first object fromthe convolutional layer of the stored artificial intelligence model andthe fully-connected layer of the stored artificial intelligence modelmay be obtained.

In addition, when the second object of the plurality of objects isidentified as present in the second area based on information on theobjects present in the second area of the plurality of areas, the secondmodel including other parts trained to identify the second object fromthe convolutional layer of the stored artificial intelligence model andthe fully-connected layer of the stored artificial intelligence modelmay be obtained.

The control method may identify the object present at each of theplurality of areas according to a pre-set interval, user input, or thelike, and use the information on the identified object to update theartificial intelligence model corresponding to each of the plurality ofareas.

Specifically, the image obtained through the camera while the electronicapparatus 100 is located at one area of the plurality of areas may beinput to the plurality of artificial intelligence models loaded in thevolatile memory and the object present at the related area may beidentified. In this case, at least one of the plurality of artificialintelligence models in the volatile memory may be sequentially loaded,and the image obtained from the related area may be input to the loadedartificial intelligence model. Based on information on the identifiedobject, the artificial intelligence model corresponding to the relatedarea may be updated.

For example, when information on objects present at each of theplurality of areas are stored in the storage of the electronicapparatus, the control method may determine at least one object presentat one area of the plurality of areas based on information on objectspresent at each of the plurality of areas stored in the storage.

In this case, based on information on the objects identified in therelated area, the object not identified in the related area from thedetermined objects may be determined. In the artificial intelligencemodel corresponding to the related area from among the plurality ofartificial intelligence models, the part trained to identify the objectdetermined previously as not identified may be removed.

In another example, when information on objects present at each of theplurality of areas is stored in the storage of the electronic apparatus100, the control method may determine at least one object present in onearea of the plurality of areas based on information on the objectspresent at each of the plurality of areas stored in the storage.

In this case, based on information on objects identified in the relatedarea, objects not included among the determined at least one object ofthe identified objects in the related area may be determined. At thistime, the fully-connected layer trained to identify the object notincluded in the at leastone determined object may be added to theartificial intelligence model corresponding to the related area fromamong the plurality of artificial intelligence models.

FIG. 15 is a diagram of an algorithm illustrating an example of acontrol method of an electronic apparatus 100 according to an embodimentupdating an artificial intelligence model corresponding to each of aplurality of areas according to results identifying an object present ineach of the plurality of areas.

Referring to FIG. 15, the control method may input an image obtainedthrough the camera in one area of the plurality of areas to at least oneof the plurality of artificial intelligence models to identify theobject (operation S1510).

Then, information on the pre-stored object of the related area may becompared with information on the identified object (operation S1520).

If information on the identified object matches with information on thepre-stored object (operation S1530-Y), the artificial intelligence modelstored to correspond to the related area may not be updated.

If information on the identified object does not match with informationon the pre-stored object (operation S1530-N), and if information on anewly added object is included in addition to the information on theidentified object with the information on the pre-stored object(operation 51540-Y), the artificial intelligence model corresponding tothe related area may be updated so that the artificial intelligencemodel stored to correspond to the related area may also identify theadded object (operation S1550).

For example, while the identified object in the related area may be a TVand a sofa, if the pre-stored object on the related area is a TV, theartificial intelligence model corresponding to the related area mayupdate the related artificial intelligence model to identify the sofa inaddition to the TV. In this case, a separate fully-connected layertrained to identify the sofa may be added to the fully-connected layerincluding the related artificial intelligence model.

Even if information on the identified object does not match with theinformation on the pre-stored object (operation S1530-N), and there isan object not included in the information on the identified objects fromthe information on the pre-stored objects (operation S1540-N), updatingthe artificial intelligence model corresponding to the related area maybe required or performed. That is, because the object present at apreviously related area is no longer present at the related area, thepart trained to identify the object no longer present in the relatedarea may be removed (operation S1560).

For example, while the identified object in the related area may be a TVand a sofa, if the pre-stored object on the related area is a TV, asofa, and a chair, the part trained to identify the chair in theartificial intelligence model corresponding to the related area may beremoved. Specifically, the part used in the inference process togenerate an output of the node indicating the possibility of the chairbeing present may be removed from the fully-connected layer of therelated artificial intelligence model. However, even if the part isrelated to the inference process for generating output of the nodeindicating the possibility of the chair being present, the part may notbe removed if the part relates to the inference process for generatingoutput of nodes indicating the possibility of the TV or the sofa beingpresent.

The control method of the electronic apparatus 100 described withreference to FIGS. 13 to 15 above may be implemented through theelectronic apparatus 100 illustrated and described above with referenceto FIGS. 2 and 11, and the electronic apparatus 100 illustrated anddescribed above with reference to FIGS. 12A and 12B.

The control method of the electronic apparatus 100 described withreference to FIGS. 13 to 15 above may be implemented through anelectronic apparatus 100 and a system including the one or more externalapparatuses.

The electronic apparatus 100 and the control method of the electronicapparatus 100 as described above may not only use all artificialintelligence models for recognizing objects of various types, but alsohas the effect of accurately recognizing objects at a fast rate.

Specifically, because the electronic apparatus 100 and the controlmethod according to one or more embodiments selectively loads onlyartificial intelligence models appropriate to the area in which theelectronic apparatus is located to use in object recognition, theelectronic apparatus 100 and the control method according to anembodiment is advantageous in that objects of many types may berecognized within a very short period of time by processing only arelatively small amount.

The electronic apparatus 100 and control method according to one or moreembodiments is also advantageous in that the artificial intelligencemodel for each area may be updated according to circumstance to quicklymaintain or further improve the object recognition rate.

The electronic apparatus 100 according to one or more embodiments maynot only be able to perform recognition of objects of a broad-rangingtype using only the self-stored artificial intelligence modelsregardless of communication with the server, but also is advantageous inthat object recognition a relatively fast rate may be possible despitethe limited processing capacity.

The various embodiments described above may be implemented in arecordable medium which is readable by computer or a device similar tocomputer using software, hardware, or the combination of software andhardware.

By hardware implementation, the embodiments described in the disclosuremay be implemented using, for example, and without limitation, at leastone of application specific integrated circuits (ASICs), digital signalprocessors (DSPs), digital signal processing devices (DSPDs),programmable logic devices (PLDs), field programmable gate arrays(FPGAs), processors, controllers, micro-controllers, microprocessors,and electrical units for performing other functions.

In some cases, embodiments described herein may be implemented by theprocessor itself. According to a software implementation, embodimentssuch as the procedures and functions described herein may be implementedwith separate software modules. Each of the above-described softwaremodules may perform one or more of the functions and operationsdescribed herein.

The computer instructions for performing the processing operations inthe electronic apparatus 100 according to the various embodimentsdescribed above may be stored in a non-transitory computer-readablemedium. The computer instructions stored in this non-transitorycomputer-readable medium cause the above-described specific device toperform the processing operations in the electronic apparatus 100according to the above-described various embodiments when executed bythe processor of the specific device.

The non-transitory computer readable medium may refer, for example, to amedium that stores data semi-permanently, rather than storing data for avery short time, such as a register, a cache, and a memory, and isreadable by an apparatus. Specific examples of the non-transitorycomputer readable medium may include a compact disc (CD), a digitalversatile disc (DVD), a hard disc, a Blu-ray disc, a universal serialbus (USB), a memory card, a read only memory (ROM), and the like.

While embodiments have been illustrated and described above, it will beunderstood by those skilled in the art that various changes in form anddetails may be made therein without departing from the true spirit andfull scope of the disclosure.

What is claimed is:
 1. An electronic apparatus, comprising: a camera; astorage for storing a plurality of artificial intelligence modelstrained to identify objects and for storing information on a map; and atleast one processor configured to: identify an artificial intelligencemodel, from among the plurality of artificial intelligence models storedin the storage based on information on an area in which the electronicapparatus is located from among a plurality of areas in the map, andinput an image obtained through the camera to the identified artificialintelligence model to identify an object.
 2. The electronic apparatus ofclaim 1, wherein: each of the plurality of artificial intelligencemodels comprises a first layer and a second layer trained to identify anobject based on characteristic information extracted from the firstlayer; the first layer is a common layer in the plurality of artificialintelligence models; and the processor is further configured to identifythe object using the first layer and the second layer of the artificialintelligence model corresponding to the area.
 3. The electronicapparatus of claim 2, wherein: the plurality of artificial intelligencemodels comprises a first model corresponding to a first area of theplurality of areas and a second model corresponding to a second area ofthe plurality of areas; and the processor is further configured to:based on the electronic apparatus being located at the first area,identify the object using the first layer and the second layercorresponding to the first model, and based on the electronic apparatusbeing located at the second area, identify the object using the firstlayer and the second layer corresponding to the second model.
 4. Theelectronic apparatus of claim 1, further comprising: a sensor, whereinthe processor is further configured to determine, based on sensing dataobtained from the sensor, the area in which the electronic apparatus islocated from among the plurality of areas.
 5. The electronic apparatusof claim 4, wherein: the information on the map comprises information onstructures of the plurality of areas; and the processor is furtherconfigured to compare the information on the structures with the sensingdata obtained from the sensor to determine the area in which theelectronic apparatus is located from among the plurality of areas. 6.The electronic apparatus of claim 4, wherein the processor is furtherconfigured to: obtain information on a structure of a place in which theelectronic apparatus is located based on sensing data obtained from thesensor, divide the place to the plurality of areas based on the obtainedinformation on the structure, generate the information on the mapcomprising information on structures of each of the divided plurality ofareas and storing the generated information on the map in the storage.7. The electronic apparatus of claim 6, wherein: the processor isfurther configured to input an image obtained through the camera whilethe electronic apparatus is located in a particular area, among theplurality of areas, to at least one artificial intelligence model amongthe stored plurality of artificial intelligence models to identify anobject included in the particular area, and obtain an artificialintelligence model corresponding to the particular area, based oninformation on the identified object.
 8. The electronic apparatus ofclaim 7, wherein: each of the stored plurality of artificialintelligence models comprises a first layer and a second layer trainedto identify a plurality of objects based on characteristic informationextracted from the first layer; and the processor is further configuredto: based on a first object of the plurality of objects in a first areabeing identified as present based on information on an object includedin the first area, obtain a first model comprising the first layer and afirst part of the second layer trained to identify the first object, andbased on a second object of the plurality of objects in a second areabeing identified as present based on information on an object includedin the second area, obtain a second model comprising the first layer anda second part of the second layer trained to identify the second object.9. The electronic apparatus of claim 1, wherein: the processor isfurther configured to input an image obtained through the camera whilethe electronic apparatus is located in the area, among the plurality ofareas, to at least one artificial intelligence model, from among theplurality of artificial intelligence models, to identify an objectincluded in the area, and update an artificial intelligence modelcorresponding to the area based on information on the identified object.10. The electronic apparatus of claim 9, wherein: the storage storesinformation on objects included in each of the plurality of areas; andthe processor is further configured to: based on the information onobjects included in each of the plurality of areas, determine at leastone object included in the area, based on information on the identifiedobject and information on the determined object, determine whether anobject different from the identified object is presented from among thedetermined object, and based on determining that the object differentfrom the identified object is presented from among the determinedobject, update the artificial intelligence model corresponding to thearea by removing a part trained for identifying the object from theartificial intelligence model corresponding to the area.
 11. Theelectronic apparatus of claim 9, wherein: the storage stores informationon objects included in each of the plurality of areas; and the processoris further configured to: based on the information on objects includedin each of the plurality of areas, determine at least one objectincluded in the area, based on information on the identified object andinformation on the determined object, determine whether the identifiedobject is different from the determined object, and based on determiningthat the identified object is different from the determined object, adda trained part for identifying the identified object to the artificialintelligence model corresponding to the area to update the artificialintelligence model corresponding to the area.
 12. The electronicapparatus of claim 11, wherein: the processor is further configured to:p1 if a plurality of objects are determined in the area based on theinformation on objects included in each of the plurality of areas,determine whether the identified object is different from each of thedetermined objects based on information on the identified object andinformation on the determined objects, and based on determining that theidentified object is different from each of the determined objects, adda trained part for identifying the identified object to the artificialintelligence model corresponding to the area to update the artificialintelligence model corresponding to the area.
 13. An electronicapparatus, comprising: a camera; a sensor; a storage for storing aplurality of artificial intelligence models trained to identify objectsand for storing information on a map; and a processor configured tocontrol the electronic apparatus, wherein the processor is configuredto: determine, based on sensing data obtained from the sensor, an areain which the electronic apparatus is located from among a plurality ofareas in the map, based on the determined area, identify an artificialintelligence model from among the plurality of artificial intelligencemodels stored in the storage, and input an image obtained through thecamera to the identified artificial intelligence model to identify anobject.
 14. A control method of an electronic apparatus using an objectrecognition model, the control method comprising: identifying aplurality of areas in a map based on information on the map stored in astorage of the electronic apparatus; identifying an artificialintelligence model, from among a plurality of artificial intelligencemodels stored in the storage, based on an area in which the electronicapparatus is located from among the plurality of areas; and identifyingan object by inputting an image obtained through a camera to theidentified artificial intelligence model.
 15. The control method ofclaim 14, wherein: each of the plurality of artificial intelligencemodels comprises a first layer and a second layer trained to identify anobject based on characteristic information extracted from the firstlayer; the first layer is a common layer in the plurality of artificialintelligence models; the second layer is a layer individually providedto each of the plurality of artificial intelligence models; and theidentifying the object comprises: identifying the object using the firstlayer and the second layer corresponding to a first model of theplurality of artificial intelligence models based on the electronicapparatus being located at a first area among the plurality of areas,and identifying the object using the first layer and the second layercorresponding to a second model of the plurality of artificialintelligence models based on the electronic apparatus being located at asecond area among the plurality of areas.
 16. The control method ofclaim 14, further comprising: determining the area in which theelectronic apparatus is located from among the plurality of areas basedon sensing data obtained from a sensor.
 17. The control method of claim16, wherein: the information on the map comprises information onstructures of each of the plurality of areas; and the determining thearea in which the electronic apparatus is located comprises comparingthe information on the structures with the sensing data obtained fromthe sensor to determine the area in which the electronic apparatus islocated from among the plurality of areas.
 18. The control method ofclaim 16, further comprising: obtaining information on a structure of aplace in which the electronic apparatus is located based on sensing dataobtained from the sensor; dividing the place to the plurality of areasbased on the obtained information on the structure; and generating theinformation on the map comprising information on structures of each ofthe divided plurality of areas and storing the generated information onthe map in the storage.
 19. The control method of claim 18, furthercomprising: inputting an image obtained through the camera while theelectronic apparatus is located in a particular area, of the pluralityof areas, to at least one artificial intelligence model among the storedartificial intelligence models, to identify an object included in theparticular area; and obtaining an artificial intelligence modelcorresponding to the particular area, based on information on theidentified object.
 20. The control method of claim 19, wherein: each ofthe stored plurality of artificial intelligence models comprises a firstlayer and a second layer trained to identify a plurality of objectsbased on characteristic information extracted from the first layer; andthe obtaining the artificial intelligence model comprises: based on afirst object of the plurality of objects in a first area beingidentified as present based on information on an object included in thefirst area, obtaining a first model comprising the first layer and afirst part of the second layer trained to identify the first object ofthe second layer, and based on a second object of the plurality ofobjects in a second area being identified as present based oninformation an object included in the second area, obtaining a secondmodel comprising the first layer and a second part of the second layertrained to identify the second object.